Debugging Daemons
Debugging daemons with GDB under Windows This is an example how to debug any daemon with GDB. I will use SLiRP as example. But it would be usable for almost all others too. Install gdb.exe * Download GDB.EXE from Mingw * Enter http://sf.net/projects/mingw/ and select all files, and than open tree "GNU Source-Level Debugger". My favoroite was "Release Candidate_ gdb-6.3" (not "Release Candidate_ GDB 6.8-3") * Download and install the exe file, for example under C:\coLinux\debug Install Process Explorer * Download ProcessExplorer from Sysinternals * Install (unzip) ProxessExplorer somethere, for example to "C:\program files\sysinternals" Get executable files with debugging informations Download version with debug symbols for GDB Skip this part, if you able to build daemons self from source. * The packages named daemons-*.dbg.zip contains the daemons with debugging symbol information. ** Download the current release daemons-0.7.6.dbg.zip from http://www.henrynestler.com/colinux/releases/ ** For branch devel please pick up the file daemons-*.dbg.zip from folder packages on http://www.henrynestler.com/colinux/testing/devel-0.8.0/ after selecting the snapshot build date, for example daemons-0.8.0-20100305.dbg.zip * Unpack all files into a separate folder, for example C:\coLinux\debug Create daemons with full debug informations Skip this part, if you have downloaded the prebuild debug version from section above. This step will give you better results (as the step above), because we enable full source and line number debugging. * Unpack source under Linux * Create configuration and build the toolchain and kernel: : (skip this, if you have it anyway) ./configure make * Change the build to not strip out debugging informations. Edit file src/colinux/os/winnt/build/build.comake.py and replace "compiler_strip = True" with "compiler_strip = False". * Set global option -ggdb and re-build the daemons: export COLINUX_CFLAGS="-ggdb" make clean make colinux * As result all daemons are mutch bigger now. * Copy the *.exe files from directory src/colinux/os/winnt/build into your Windows directory, for example to C:\coLinux\debug * Caution 1: The files in build directory are soft links. So, you needs to use cp -L ... (and not cp -a), or other options like "follow links" in your copy-tool. * Caution 2: Only use *.exe files. Never use linux.sys from this debugging build! This will crash with blue screen. It is a problem with unresolved external debugging symbols from GCC libc. * What to do in case of BSOD: in case you did copy the file linux.sys and wish to restart, you will notice that your Windows keeps getting blue screens. To fix this, enter Windows in safe mode and manually delete the file linux.sys with the debugging information (typically in c:\Program Files\coLinux). : Unpack source under Windows To see the source later in debug session you need to extract sources under Windows. The directory name src must start in current directory. * Download the source for your running version. ** Source for releases you can get from SF, for example the file coLinux-0.7.6-src.tar.gz, or from here ** Source for branch devel you will find under Snapshots as file devel-colinux-*.tar.gz * Unpack coLinux sources to any directory * Move the directory src into debugging directory C:\coLinux\debug In this example here, the full path to source is C:\coLinux\debug\src/colinux/user/slirp/tcp_input.c. Start daemon with gdb *Run coLinux with your typical configuration and the default executables *Run ProcessExplorer **Locate the task colinux-daemon.exe. Write down the PID (process ID) for this task. The PID is an arbitrary number and will vary every time you run coLinux, for example, 2496. **Locate the task you want to debug, for example colinux-slirp-net-daemon.exe and kill (terminate) this task followed with a "Yes" answer to terminate the process. * Open a Windows command prompt (Start -> Run... -> CMD.EXE) and change the directory to the debugging installation. It is also wise to add the bin directory of MingW debugger to the PATH variable. Run GDB with the daemon there: cd C:\coLinux\debug path=%path%;c:\mingw\bin gdb -se colinux-slirp-net-daemon.exe * On the GDB-Prompt run daemon with parameters "-i 2496 -u 0" you have in your clipboard: (gdb) run '-i 2496 -u 0' * Now the daemon is running under the debugger: Starting program: C:\coLinux\debug/colinux-slirp-net-daemon.exe -i 5244 -u 0 conet-slirp-daemon: running Get informations after crash Mostly you want to see why a daemon is crashing, you needs now do some that will cause the effect to crash or end the daemon. * After a daemon has crashed you will see the address and line number, for example: Program received signal SIGSEGV, Segmentation fault. tcp_input (m=0xaba528, iphlen=20, inso=0xe07ef8) at src/colinux/user/slirp/tcp_input.c:1460 1460 if (ti->ti_len && (unsigned)ti->ti_len <= 5 && Please note this lines and send to developers. (Use left mouse menu and copy it into a mail.) * Please dump out the stack with bt (gdb) bt #0 tcp_input (m=0xaba528, iphlen=20, inso=0xe07ef8) at src/colinux/user/slirp/tcp_input.c:1460 #1 0x00402b90 in tcp_input (m=0x9ca850, iphlen=2358284, inso=0x23fb08) at src/colinux/user/slirp/tcp_input.c:1403 #2 0x00402b90 in tcp_input (m=0x40cc50, iphlen=206528, inso=0x23ff68) at src/colinux/user/slirp/tcp_input.c:1403 ... Send this lines also please to developers. * Additional for the failed line 1460 above, it would be interesting which variable is the problem. So try to get values from some variables you can see in breaking line 1460, for example: (gdb) print ti $1 = (struct tcpiphdr *) 0xaba55c (gdb) print ti->ti_len There is no member named ti_len. "ti_len" is a macro for "ti_i.ih_len". So we need do this: (gdb) print ti->ti_i.ih_len Cannot access memory at address 0xaba566 (gdb) print ti->ti_i Cannot access memory at address 0xaba55c In this example we have a problem with ti->ti_i.ih_len and ti->ti_i. Now you can fix the problem, or send your results to developers or add your gdb session as log to coLinux's BugTracker system.